Evaluating a Music Information Retrieval System - TREC Style
نویسندگان
چکیده
Recent efforts made by the MIR community is pushing for the creation of a standardised framework for MIR system evaluation. Such a framework will allow the community to collaborate on important issues and ultimately progress this dynamic area of research further. Some of these efforts are asking if a framework similar to TREC is the way forward. In this paper we report the methodology and results of a TREC style evaluation of a music information retrieval system called Ceolaire.
منابع مشابه
Whither Music IR Evaluation Infrastructure: Lessons to be Learned from TREC
Benchmark tasks are a powerful method for advancing the state of the art in a field. The music information retrieval community recently acknowledged the utility of such tasks by resolving to create an evaluation framework for music information retrieval (MIR) tasks and music digital libraries (MDL). This paper describes the processes used in the Text REtrieval Conference (TREC) evaluations to c...
متن کاملAn Initial Test Collection for Ranked Retrieval of SMS Conversations
This paper describes a test collection for evaluating systems that search English SMS (Short Message Service) conversations. The collection is built from about 120,000 text messages. Topic development involved identifying typical types of information needs, then generating topics of each type for which relevant content might be found in the collection. Relevance judgments were then made for gro...
متن کاملPrototyping a Vibrato-Aware Query-By-Humming (QBH) Music Information Retrieval System for Mobile Communication Devices: Case of Chromatic Harmonica
Background and Aim: The current research aims at prototyping query-by-humming music information retrieval systems for smart phones. Methods: This multi-method research follows simulation technique from mixed models of the operations research methodology, and the documentary research method, simultaneously. Two chromatic harmonica albums comprised the research population. To achieve the purpose ...
متن کاملAudio-Indexing For Broadcast News
In this paper we describe the IBM Audio-Indexing System which is a combination of a large vocabulary speech recognizer and a text-based information retrieval system. Our speech recognizer was used to produce the baseline transcripts for the NIST SDR97 evaluation. We report the performance of the system on the SDR-97 \known item retrieval" task and on a more pertinent TREC-style Audio-Indexing
متن کاملMethods for Evaluating Interactive Information Retrieval Systems with Users by Diane
Methods for Evaluating Interactive Information Retrieval Systems with Users by Diane Kelly is the first textbook to extensively focus on the methodologies of user studies in Information Retrieval (IR). This 224-pages book consists of 15 chapters covering a range of topics from the development of cognitive aspects in IR, to the foundation of research, to IR test collections, to the ethical issue...
متن کامل